Bayesian Distributional Policy Gradients

Distributional Reinforcement Learning (RL) maintains the entire probability distribution of reward-to-go, i.e. return, providing more learning signals that account for uncertainty associated with policy performance, which may be beneficial trading off exploration and exploitation in general. Previous works distributional RL focused mainly on computing state-action-return distributions, here we model state-return distributions. This enables us to translate successful conventional algorithms are based state values into RL. We formulate Bellman operation as an inference-based auto-encoding process minimises Wasserstein metrics between target/model return The proposed algorithm, BDPG (Bayesian Policy Gradients), uses adversarial training joint-contrastive estimate a variational posterior from returns. Moreover, can now interpret prediction information gain, allows obtain new curiosity measure helps steer actively efficiently. demonstrate suite Atari 2600 games MuJoCo tasks, including well known hard-exploration challenges, how learns generally faster higher asymptotic performance than reference algorithms.